Orchestrator: multi-repo polling for asm_* module CI - #400
Merged
Conversation
…module repos The spawn loop gathers queued jobs across every configured repo (JIT runner configs minted per-repo), state entries record their repo for reaping, and the orphan sweep / max_concurrent cap span the whole set. The legacy singular repo: key is still accepted; pre-multi-repo state entries reap against the first listed repo. config.example.yaml lists trunk plus the three asm_* module repos; README's run-a-second-instance procedure is replaced by the repos: list. Bump version to 0.20.0-alpha.10. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Test Metrics —
|
…> 0.20.0-alpha.19
Contributor
Test Metrics —
|
| Test | Pass | Fail | Skip | Rate |
|---|---|---|---|---|
| test_colcon_build_gcs | 1 | 0 | 0 | 100% |
| test_colcon_build_ms_airsim | 1 | 0 | 0 | 100% |
| test_colcon_build_robot | 1 | 0 | 0 | 100% |
| test_colcon_test_robot | 1 | 0 | 0 | 100% |
Metrics
| Test | Metric | Value |
|---|---|---|
| test_colcon_build_robot | duration_s | 87.64s |
| test_colcon_test_robot | duration_s | 39.6s |
| test_colcon_build_gcs | duration_s | 68.31s |
| test_colcon_build_ms_airsim | duration_s | 15.11s |
andrewjong
added a commit
that referenced
this pull request
Aug 29, 2026
…ence docs Scope cut agreed with Andrew before merge: - Drop tests/system/test_optitrack_e2e.py: it sets LAUNCH_NATNET (a removed no-op shim) and targets robot/ros_ws/src/perception/natnet_ros2, which was extracted from trunk (70423c4). It could never pass on develop and PR CI never executes the optitrack mark, so it would rot silently. The same test lives where it belongs: asm_optitrack/tests/system/, exercised by the module CI that is now served by the multi-repo orchestrator (#400). The mark stays registered in tests/pytest.ini for module-CI runs. - Drop tests/report-requirements.txt: develop's metrics job already installs tests/requirements.txt (#407); a second requirements file for the same tree is a drift hazard. The workflow contract now pins the #407 behavior. - Drop harness/image_prep.py and the image-preparation.json workflow instrumentation: metadata plumbing whose value doesn't carry its weight — the associative-array bash was the most fragile code in the PR. The image-prep step returns to develop's plain version. - diagnostics: drop LAUNCH_NATNET from SAFE_ENV_KEYS (dead); keep PX4_PARAM_SET (live compose env-file selector). - Un-clobber develop content the branch predated: run-system-tests SKILL description (waypoint_flight, wiring marks), BSD-3-Clause-Clear license, `airstack images build` spelling. - Reference docs refreshed to where CI stands now: AGENTS.md test-suite paragraph (schema-v2 run_meta, advisory metric deltas, diagnostics bundle, fingerprint-only comparisons) and --config-only intent flag; module_ci.md + module-system-tests.yml header (orchestrator polls a repos: list — asm_optitrack included today); Release Notes entry for the trustworthy-outcomes policy. Full unit suite on this branch: 419 passed, 0 failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
andrewjong
added a commit
that referenced
this pull request
Aug 29, 2026
* Make system-test outcomes trustworthy and actionable Separate infrastructure failures, test assertions, and advisory metric changes so CI blocks only on real test or integrity failures while preserving comparable performance evidence. * Classify readiness failures as infrastructure Preserve explicit infrastructure intent through pytest call reports so simulator startup crashes cannot be misreported as algorithm assertions. * Allow focused manual flight campaigns Expose trajectory and takeoff sweeps in workflow dispatch so CI validation can run minimal algorithm samples before expanding to expensive matrices. * Finish leftover merge markers from the develop rebase. Keep develop's images CLI and Using CI layout while preserving this PR's advisory metrics, diagnostics artifacts, and infrastructure mark. * Fix unit-test merge fallout and bump VERSION. Keep --config-only hermetic for fleet contracts, still fail AUTONOMY_ROLE as a config check, and document the new help tokens so CI can pass the increment gate. * Strip post-extraction residue and metadata plumbing; refresh CI reference docs Scope cut agreed with Andrew before merge: - Drop tests/system/test_optitrack_e2e.py: it sets LAUNCH_NATNET (a removed no-op shim) and targets robot/ros_ws/src/perception/natnet_ros2, which was extracted from trunk (70423c4). It could never pass on develop and PR CI never executes the optitrack mark, so it would rot silently. The same test lives where it belongs: asm_optitrack/tests/system/, exercised by the module CI that is now served by the multi-repo orchestrator (#400). The mark stays registered in tests/pytest.ini for module-CI runs. - Drop tests/report-requirements.txt: develop's metrics job already installs tests/requirements.txt (#407); a second requirements file for the same tree is a drift hazard. The workflow contract now pins the #407 behavior. - Drop harness/image_prep.py and the image-preparation.json workflow instrumentation: metadata plumbing whose value doesn't carry its weight — the associative-array bash was the most fragile code in the PR. The image-prep step returns to develop's plain version. - diagnostics: drop LAUNCH_NATNET from SAFE_ENV_KEYS (dead); keep PX4_PARAM_SET (live compose env-file selector). - Un-clobber develop content the branch predated: run-system-tests SKILL description (waypoint_flight, wiring marks), BSD-3-Clause-Clear license, `airstack images build` spelling. - Reference docs refreshed to where CI stands now: AGENTS.md test-suite paragraph (schema-v2 run_meta, advisory metric deltas, diagnostics bundle, fingerprint-only comparisons) and --config-only intent flag; module_ci.md + module-system-tests.yml header (orchestrator polls a repos: list — asm_optitrack included today); Release Notes entry for the trustworthy-outcomes policy. Full unit suite on this branch: 419 passed, 0 failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Pranav Kumara <pkumara@andrew.cmu.edu> Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One orchestrator instance now serves trunk and the module repos:
config.yamltakes arepos:list (legacyrepo:still accepted), the spawn loop collects queuedairstack-ephemeraljobs across all of them with per-repo JIT runner minting, and reaping/orphan-sweep/max_concurrentspan the set. Unblocks GPU CI for asm_dfm2_disturbances / asm_optitrack / asm_macvo (theirci.ymlcallers now pin the reusable workflow@develop).Host-side rollout (orchestrator machine): pull this commit, switch
/etc/airstack-orchestrator/config.yamlto therepos:list, extend the PAT to the three module repos (Actions + Administration read/write),systemctl restart airstack-orchestrator, and confirm the startup line showsrepos=[...].Unit suite: 400 passed / 7 skipped. VERSION → 0.20.0-alpha.10; Release Notes updated.
🤖 Generated with Claude Code